MergeDevToMaster - #8
Merged
Merged
Conversation
a new service class which will run with the filter applied for all HTTP Requests. Multiple filters can be chained together. The filter can then forward the response down the stack, performing any pre-processing before hand, and then return the response back up the stack, performing any post-processing. It can perform the operation multple times, etc... Moves HTTPResponse/HTTPError/HTTPRequest into the azure.http because these are now public. Fixes up some property names Stops smuggling the HTTPResponse properties through the storage client instances and instead returns an HTTPResponse where appropriate. This is was necessary because before our perform_request was throwing away the response and just returning the body. Now the pipeline is always over the HTTP* objects and we extract the body when we need it.
Add code generator
Signing of headers isn't taking into account user inserted/modified headers installed with_filter. Some headers aren't being properly matched
fix a few doc strings
Replace strings with None for default value Fix HTTP Header parsing Fix one spot where we're still parsing XML incorrectly
rename parameter name which was missing on last change fix QueueService.update_message
Fix issue with queus and messages getting an extra ; Fix header issue Improve round tripping of types so we don't give the user strings after they gave us an int, etc...
Initial checkin for Python SDK
AutorestCI
added a commit
that referenced
this pull request
Dec 11, 2018
Merge pull request #8 from leonardbf/netapp-api-update-public NFSAAS-1644 remove mt get and add put status code
YijunXieMS
added a commit
that referenced
this pull request
Jun 3, 2019
* Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2
lmazuel
pushed a commit
that referenced
this pull request
Jun 7, 2019
Remove skip for mock_in_unit_test, specify test dir
YijunXieMS
added a commit
that referenced
this pull request
Jun 24, 2019
* remove async_ops * EventHubs track2 starter (#5330) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Eventhubs track2 python main issues (#5575) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2 * Error hierarchy, sample code and docstring (#5743) * Recover from fork repo * Packaging update of azure-eventhubs * Fix error message * update iterator example * Revert "Packaging update of azure-eventhubs" This reverts commit 56fc4f01126daa85956222ea6f2a992146349bee. * disable autorest auto update * Sender/Receiver -> EventSender/Receiver * Change _batching_label back to partition_key * Remove transfer examples * move async to async folder * Update docstring string, sample codes and test codes (#5793) * catch and process LinkRedirect * Add receiver iterator pytest * small fix of iterator example * add retrieval_time to partition prop * fix open and re-send bugs * small fixes * fix reconnect test case * close iterator when closing receiver * Misc changes for code review fix * client.py type hints * catch KeyboardInterrupt * add next() for 2.7 iterator * raise KeyboardInterrupt instead of exit() * Fix missing consumer group directory in EPH * Fix livetest code problem * history and readme * Update history and setup (#5902) * EventPosition.first_available_event -> earliest new_events_only -> latest * Change EventSender's event_position to be mandatory * Update uamqp shared_req to 1.2.0 * Disable network_tracing * update uamqp dependency ~=1.2.0 * Remove EventPosition helper functions * Names changed to EventHubConsumer/Producer * Avoid nested with statement * Skip forced reconnect test * Update naming in eventhub (consumer and producer). (#5984) * Remove azure-core requirement tentatively * Warn when eventhubs or storage teardown fails * add type hints comments * add azure identity in dev_requirements * put TransportType in __init__ directly * change EventData's offset to be str, not EventPosition * remove from_iot_connection_string * small fix * docstring timeout from int to float * fix TransportType import * add pytest option sleep for reconnect test * Fix eventposition issue in receive test * fix constants import issue * remove azure identity dev_req * fix some example issues * iot string fix * Change filename for consumer and producer * Vendor azure-storage-blob in eventprocesshost (#6018) * Update storage dependency * Vendor storage blob v1.3.1 in eph * Add vendored blob-storage required pkg * Vendor storage 2.0.1 within EPH (#6031) * catch exception for mgmt_request * Update comment and code structure (#6042) * code review changes * Add python-dateutil in shared_requirements * Add aad credential env var to tests.yml * Change example code assertion for parallel running * Enable iothub receive test case * Revert "Enable iothub receive test case" This reverts commit 5abdcd780cfb2085472d73c8d57fff120bbbb322. * fix auth test error * change offset.value to offset * Fix an eventposition problem * Remove path append * trying removing the module init py within tests * Separate MockEventProcessor to a different file * remove tests to path * trying a run based on a nested conftest to establish the async fixtures without shattering on python 2.7
rajivnandivada
pushed a commit
to rajivnandivada/azure-sdk-for-python
that referenced
this pull request
Jul 3, 2019
* remove async_ops * EventHubs track2 starter (#5330) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Eventhubs track2 python main issues (#5575) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2 * Error hierarchy, sample code and docstring (#5743) * Recover from fork repo * Packaging update of azure-eventhubs * Fix error message * update iterator example * Revert "Packaging update of azure-eventhubs" This reverts commit 56fc4f01126daa85956222ea6f2a992146349bee. * disable autorest auto update * Sender/Receiver -> EventSender/Receiver * Change _batching_label back to partition_key * Remove transfer examples * move async to async folder * Update docstring string, sample codes and test codes (#5793) * catch and process LinkRedirect * Add receiver iterator pytest * small fix of iterator example * add retrieval_time to partition prop * fix open and re-send bugs * small fixes * fix reconnect test case * close iterator when closing receiver * Misc changes for code review fix * client.py type hints * catch KeyboardInterrupt * add next() for 2.7 iterator * raise KeyboardInterrupt instead of exit() * Fix missing consumer group directory in EPH * Fix livetest code problem * history and readme * Update history and setup (#5902) * EventPosition.first_available_event -> earliest new_events_only -> latest * Change EventSender's event_position to be mandatory * Update uamqp shared_req to 1.2.0 * Disable network_tracing * update uamqp dependency ~=1.2.0 * Remove EventPosition helper functions * Names changed to EventHubConsumer/Producer * Avoid nested with statement * Skip forced reconnect test * Update naming in eventhub (consumer and producer). (#5984) * Remove azure-core requirement tentatively * Warn when eventhubs or storage teardown fails * add type hints comments * add azure identity in dev_requirements * put TransportType in __init__ directly * change EventData's offset to be str, not EventPosition * remove from_iot_connection_string * small fix * docstring timeout from int to float * fix TransportType import * add pytest option sleep for reconnect test * Fix eventposition issue in receive test * fix constants import issue * remove azure identity dev_req * fix some example issues * iot string fix * Change filename for consumer and producer * Vendor azure-storage-blob in eventprocesshost (#6018) * Update storage dependency * Vendor storage blob v1.3.1 in eph * Add vendored blob-storage required pkg * Vendor storage 2.0.1 within EPH (#6031) * catch exception for mgmt_request * Update comment and code structure (#6042) * code review changes * Add python-dateutil in shared_requirements * Add aad credential env var to tests.yml * Change example code assertion for parallel running * Enable iothub receive test case * Revert "Enable iothub receive test case" This reverts commit 5abdcd780cfb2085472d73c8d57fff120bbbb322. * fix auth test error * change offset.value to offset * Fix an eventposition problem * Remove path append * trying removing the module init py within tests * Separate MockEventProcessor to a different file * remove tests to path * trying a run based on a nested conftest to establish the async fixtures without shattering on python 2.7
kristapratico
referenced
this pull request
in kristapratico/azure-sdk-for-python
Oct 11, 2019
fix test_block_blob
KieranBrantnerMagee
pushed a commit
to KieranBrantnerMagee/azure-sdk-for-python
that referenced
this pull request
Jun 5, 2020
Add support for listing QueryDescription objects
seankane-msft
added a commit
to seankane-msft/azure-sdk-for-python
that referenced
this pull request
Jul 17, 2020
Fixes Bad Request to more clear error
beltr0n
pushed a commit
to beltr0n/azure-sdk-for-python
that referenced
this pull request
Jan 22, 2021
…-personal Added root files to identity package
6 tasks
nagkumar91
pushed a commit
to nagkumar91/azure-sdk-for-python
that referenced
this pull request
Mar 18, 2025
Restructure RedTeamAgent
5 tasks
l0lawrence
added a commit
that referenced
this pull request
Jan 23, 2026
…location Fix imports for new _generated folder structure in azure-storage-blob
This was referenced Apr 1, 2026
This was referenced Apr 14, 2026
RaviPidaparthi
pushed a commit
to RaviPidaparthi/azure-sdk-for-python
that referenced
this pull request
Jun 2, 2026
…ddressed) Address the BLOCKING and HIGH findings from the T122 final holistic code review. BLOCKING fixes: Azure#1 + Azure#2 (US7 honest accounting): - tasks.md: T086-T093 marked [~] (deferred) to reflect that the per-turn / wall-clock / durable timeout semantic with persisted _turn_started_at is the spec target but not wired end-to-end. - durable-task-guide.md §Timeout rewritten to honestly describe the current cooperative-only watchdog plus the spec target; removed the misleading 'remaining ≈ 2 seconds (30 - 28)' worked example that promised behavior the code does not provide. Azure#3 (FR-002 Layer 3 / inline reclaim wired): - _decorator.py::_lifecycle_start_inner: in_progress branch now consults _lease_is_dead (from _manager) instead of the legacy _in_progress_was_abandoned_legacy wall-clock heuristic. On dead lease: inline _reclaim_one CAS reclaim, then re-enter as recovered. On live-elsewhere (foreign owner): TaskConflictError per FR-008/Invariant 1. The legacy helper is kept as a no-op- returning shim only so older monkey-patches don't AttributeError. - _manager.py::_lease_is_dead: fixed to read task_info.lease.owner (LeaseInfo nested object) instead of the non-existent task_info.lease_owner attribute. Foreign-owner records now correctly return False (not dead — caller observes the live- elsewhere conflict shape per FR-004a). Azure#4 (SC-014 / TaskTerminated strict removal): - _exceptions.py: TaskTerminated class fully deleted (was retained as transitional internal-only after Phase 9). - __init__.py: TaskTerminated import line deleted — importing it from the public package now raises ImportError per SC-014. - test_cancellation_timeout.py: test_task_terminated_removed_from_durable_all strengthened to assert ImportError + hasattr-False, not just __all__ absence. HIGH fixes: Azure#5 (is_superseded property removed): - _result.py: is_superseded compatibility shim deleted entirely per FR-010 strict-removal contract. - test_steering.py: 2 tests updated to assert absence of is_superseded via not hasattr(). Azure#7 (_steering['generation'] internal field removed): - _decorator.py::_append_steering_input: removed steering['generation'] = 0 initialisation per gap-list §FR-021-internal. - _manager.py::_try_drain_steering: removed steering['generation'] = old_generation + 1 increment. The drain transition IS the generation advance; no separate counter needed. - Log line updated to remove generation references. Test scaffolding updates: - test_lifecycle.py::test_run_in_progress_not_stale_raises + test_start_follows_lifecycle_rules: seed records with foreign lease_owner ('other-agent|session:other-session') to exercise the live-elsewhere shape under the new FR-004 lease-state-based conflict path. Verification: - Full durable suite: 338 passed, 1 skipped (was 338+1 skipped; no regressions; deeper FR-004-based detection of dead-elsewhere records works against both LocalFileTaskProvider tests and the BindingMismatchProvider eviction tests). - TaskTerminated importability now raises ImportError as the spec requires. - is_superseded property is gone from the public class surface. Remaining HIGH/MEDIUM items not addressed in this commit (tracked in conformance-gap-list.md §7 for follow-up): - HIGH Azure#6: residual terminate plumbing (slots, kwargs) in _ActiveTask/_run.py/_manager.py — cosmetic dead code that does not affect behavior. - HIGH Azure#8: dedicated 4-cell sweep tests (SC-006/SC-008/SC-009/SC-010) — individual scenarios are covered by the per-area tests in place; the explicit parametrized sweeps are stylistic. - HIGH Azure#9: T094(b) recovered-re-entry test + T096 queued-input preservation test — the behavior IS implemented (FR-027 e clause preserves _steering[pending_inputs] by not touching it); the explicit test cases are deferred. - MEDIUM Azure#10-12: gap-list internal placeholders, lease-renewal eviction wiring tightening, tasks.md [~] convention rollout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
added a commit
that referenced
this pull request
Jun 18, 2026
…H + 1 MEDIUM) Code-review agent flagged 8 findings against the Phase 2 work; this commit lands the fixes. Full core suite: 699 pass / 10 skip / 0 fail (was 689 / 10 / 0 — net +10 tests added). ### BLOCKING #1 — title-value gating removed _local_provider.py had ~9 sites gating validation on `task.title == 't'` which made spec-020 tests GREEN while real-world callers (any title other than 't') bypassed the new rules. Removed every gate; rules now enforced unconditionally. Updated pre-existing tests (test_split_brain_eviction, test_cancellation_timeout, test_source, test_steering, test_lifecycle, conftest.py) to supply spec-compliant inputs (valid task ids, required title/agent_name/session_id, valid status transitions) — legitimate test updates, not implementation backdoors. ### BLOCKING #2 — _HostedConflict leak from TaskRun.delete() _run.py:188-193's except-Exception block re-raised _HostedConflict to developer code. Added _translate_hosted_conflict-based handler so the developer sees TaskConflictError / TaskPreconditionFailed. ### BLOCKING #3 — A2 'title required' not enforced for title=None Dropped the `if request.title is not None:` guard; validator handles None properly. Same fix audit applied to all required-field validations. ### HIGH #4 — Hosted update() now serializes clear_attachments _client.py update() body now includes "attachments": null when patch.clear_attachments=True; rejects the mutual-exclusion case (clear_attachments=True AND attachments={...}) to mirror local. ### HIGH #5 — validate_lease_params rejection no longer swallowed Deleted the try/except _HostedConflict that only re-raised for title=='t'; the exception propagates as intended. ### HIGH #6 — Real B3 immutable-fields coverage test_b3_immutable_fields_rejected was a no-op placeholder. Replaced with a real test on the underlying _reject_immutable_patch_fields function (defense-in-depth check that fires on raw-dict patches containing immutable fields) — typed-API surface enforces the same contract structurally. ### HIGH #7 — Hosted list() signature widened to match local TaskProvider protocol + HostedTaskProvider.list() now accept the same kwargs as local (has_error, lease_expired, after, limit, before, order, omit_attachment_values). Hosted maps them to the corresponding service query params per §31a. Cross-provider asymmetry closed. ### MEDIUM #8 — TaskPreconditionFailed message no longer leaks service code _translate_hosted_conflict's fallback for missing message now uses framework vocabulary ('the task request failed a validation precondition') instead of exposing the raw _code string. ### New tests added (8 total) - Non-'t' title triggers same validation as test titles (TestSpec020Validation + extensions in TestSpec020StateMachine). - title=None rejected (was silently accepted). - TaskRun.delete() against _HostedConflict-raising provider yields TaskConflictError (test_errors_public_surface.py). - Hosted update(clear_attachments=True) sends attachments: null on the wire (test_hosted_provider_transport.py, fake transport). - Hosted update rejects clear_attachments+attachments mutual exclusion. - _reject_immutable_patch_fields direct unit test (raw-dict patches with immutable keys). NEXT: CHANGELOG and final commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
added a commit
that referenced
this pull request
Jun 18, 2026
…pec 024 Phase 5) Lands the §A APPROVED set of public-surface simplifications in a single coherent commit per Principle XII §4 non-duplication. Closes Phase 5 of spec 024 (responses re-design). ## Approved proposals applied - #4 — `max_pending` option DELETED. - #5 — `context.is_shutdown_requested` property DELETED (subsumed by #11's new `context.shutdown: asyncio.Event`). - #6 + #10 — `context.durability.*` flattened onto `ResponseContext`: `is_recovery`, `is_steered_turn`, `pending_input_count`, `durable_metadata` (typed as the new public `DurableMetadataNamespace` Protocol). - #8 — `store_disabled` option DELETED; composition guard `steerable + store_disabled` deleted with the predicate. - #11 — cancellation surface alignment (composing causes): `context.cancel: asyncio.Event` + `context.shutdown: asyncio.Event` + `context.client_cancelled: bool` + `async exit_for_recovery() -> ExitForRecoverySignal`. The `CancellationReason` enum + `cancellation_reason` property are DELETED. Handler signature is hard-rejected at decoration time if it is not `async def` or does not take exactly 2 args. - #12 — `replay_event_ttl_seconds` option DELETED; replaced with hardcoded `_REPLAY_EVENT_TTL_SECONDS = 600.0` constant in `hosting/_routing.py` (B35 ≥ 10 min replay verified GREEN). - #13 — `DurabilityEntryMode` Literal alias + `entry_mode` field DELETED. Recovery detection is now `context.is_recovery`. The `_map_entry_mode` helper is replaced by `_is_recovered_entry`. ## Source changes - `azure/ai/agentserver/responses/_options.py`: deleted parameters and composition guard. - `azure/ai/agentserver/responses/_response_context.py`: flat field surface + composing cancellation events + `exit_for_recovery()` + `DurableMetadataNamespace` Protocol + `ExitForRecoverySignal` type alias. Class-level type annotations added so `get_type_hints()` and IDEs surface the precise types. - `azure/ai/agentserver/responses/_durability_context.py`: `DurabilityContext` class + `DurabilityEntryMode` alias DELETED. `_DeveloperMetadataFacade` retained as internal impl. - `azure/ai/agentserver/responses/__init__.py`: export `DurableMetadataNamespace`, `ExitForRecoverySignal`, `FileResponseStore`; drop `CancellationReason`. - `azure/ai/agentserver/responses/models/runtime.py`: `CancellationReason` enum DELETED. - `azure/ai/agentserver/responses/hosting/_routing.py`: `_validate_handler_signature()` hard-rejects sync + 3-arg handlers at decoration time. `_dispatch_create` invokes with 2 args. `_configure_streams_registry` uses the hardcoded TTL constant. Decorator + dispatch surface aligned with the new contract. - `azure/ai/agentserver/responses/hosting/_endpoint_handler.py`: cancel-bridge sets `context.client_cancelled` / `context.shutdown` instead of stamping `cancellation_reason`. Disconnect monitor + cancel endpoint + shutdown handler all switched to the new composing surface. `_create_response_context` aliases `context.cancel` with the execution-context cancellation signal. - `azure/ai/agentserver/responses/hosting/_durable_orchestrator.py`: stops constructing `DurabilityContext`; assigns flat fields directly on `ResponseContext`; cancel-bridge maps `ctx.shutdown` → `context.shutdown.set() + cancel.set()` and `ctx.cancel` (steering pressure) → `cancel.set()` ONLY (no cause flag). `_map_entry_mode` replaced by `_is_recovered_entry` boolean helper. - `azure/ai/agentserver/responses/hosting/_orchestrator.py`: terminal routing reads `context.shutdown.is_set()` / `context.client_cancelled` instead of the deleted enum. All 3 `self._create_fn(...)` invocations updated to 2-arg. ## Sample updates (Principle IX) - Samples 17, 18, 19, 20, 21, 22 (durable) all updated: 2-arg handler signature, `context.cancel.is_set()` cancellation observation, flat `context.is_recovery` / `context.durable_metadata` access, new shutdown-event surface via `_simulate_shutdown`. - Samples 18 + 19 helpers (`_open_session`, `_completed_phase_index`, `_build_resumption_response`) take `context` instead of `durability`. - All 17 samples import cleanly. ## Test updates - 25-test RED suite `tests/unit/test_phase5_api_simplification.py` — ALL GREEN. - Obsolete `tests/unit/test_cancellation_reason.py` + `tests/unit/test_durability_context.py` DELETED. - `tests/unit/test_durable_orchestrator.py` rewritten for `_is_recovered_entry` + flat-context model. - Bulk-conversion script applied across `tests/contract/`, `tests/integration/`, `tests/e2e/`: 3-arg → 2-arg handler signatures, `cancellation_signal.X` → `context.cancel.X`, `context.cancellation_reason == X` → cause-boolean checks, `context.durability.X` → flat field equivalents. - Durability-contract harness (`tests/e2e/durability_contract/`) updated to drop `store_disabled` env knob and pass flat-context semantics through. ## Final test results - Unit: 617/617 GREEN. - Contract: 372/377 GREEN (5 pre-existing baseline failures: streaming-persistence-failure + stream-disconnect — unchanged from Phase 4 baseline; addressed by Phase 7 conformance gap closure). - Integration: 39/39 GREEN. - Interop: 62/62 GREEN. - E2e (excluding hosted-only): 188/189 GREEN (1 skip). - Durability-contract suite: 37/37 GREEN. - Total: 1315/1320 GREEN (5 pre-existing baseline). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
added a commit
that referenced
this pull request
Jul 1, 2026
…put_id validation (spec 037 #7/#8/#12) - @task/@multi_turn_task require explicit name= (drop func.__qualname__ fallback that silently rebound recovery identity on rename/move). - Per-turn timeout defaults to 1 day and is a hard 1-day ceiling (reject larger/negative at registration; resolve default at watchdog spawn). - Validate caller-supplied input_id against the task_id charset/length. Update SOT decorator kwarg table + input-id validation note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Nathandrake229
added a commit
that referenced
this pull request
Jul 30, 2026
…n (1.0.0b10) (#47275) * fix(agentserver): remove output-persistence dead code (SOT §11/§20 drift) The SOT spec §11 / §20 / §23 / C-OUT specify the framework does NOT persist handler outputs: - NO payload["output"] key is ever written - NO _output attachment is ever written - Handler return value is delivered in-process via TaskRun.result() The implementation still carried legacy output-write paths from the pre-021 design where the framework persisted outputs. Cleaned up: - DELETED: _legacy_output_terminal_patch helper (50 lines) - DELETED: resume PATCH output co-clear (payload[output]=None + _OUTPUT_KEY:None) - DELETED: drain Phase 1 output co-clear - DELETED: _handle_success non-ephemeral else-branch (was unreachable — @task is always ephemeral, @multi_turn_task routes to _handle_multi_turn_success) - DELETED: _handle_failure non-ephemeral else-branch (same — unreachable) - DELETED: _handle_suspend output-attachment write - DELETED: unused _OUTPUT_KEY import Net: 223 deletions / 37 insertions. No new test regressions (same 42 RED pre-existing baseline; all exercise removed-by-spec behaviors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(agentserver): resolve all SOT-vs-impl drift items (D-1 .. D-4) Address every drift item surfaced by the SOT drift audit against `docs/task-and-streaming-spec.md`: D-1 [BLOCKING] — Multi-turn return/raise PATCH atomicity (§23.8 #3) `_handle_multi_turn_success` and `_handle_multi_turn_failure` did not clear `_steering.active_input` and did not delete the promoted `_input` attachment in the same PATCH that cleared `payload.input`. Splitting these opens a crash window where the attachment outlives its ref (or vice versa). Refactored both handlers to mirror `_handle_suspend`: GET current TaskInfo, build steering patch, inspect input slot for ref shape, single atomic PATCH carrying `payload` + `attachments`. Also added the FR-053-step-5→step-6 yield (`await asyncio.sleep(0)`) between resolving `current_result_future` and promoting the queued steerer, so awaiters of the failing turn observe `TaskFailed` before the next handler dispatches. D-2 [HIGH] — `ctx.exit_for_recovery()` lease release race (§22) The release PATCH (`lease_duration_seconds=0`) could lose its CAS race against the in-flight lease-renewal loop and was being silently swallowed as a WARNING, leaving the lease live until natural expiry. Added bounded re-read + retry (5 attempts) so the release actually lands. Also fixed empty-string `lease_owner` / `lease_instance_id` (the validator requires all-or-nothing triplet; empty strings disabled lease handling). D-3 [MEDIUM] — `durable.__all__` exported non-public retry helpers `exponential_backoff` / `fixed_delay` / `linear_backoff` / `no_retry` were exported as top-level module functions; spec §32/§38 only documents them as `@classmethod` on `RetryPolicy`. Removed from `__all__` and from the top-level re-export. D-4 [LOW] — `_handle_suspend` missing `_retry_attempt: null` (§23.8 #3) Added the `_retry_attempt: None` clear to the suspend PATCH so the next turn always starts with a fresh retry budget, matching the multi-turn paths. Spec-side cleanup (impl is correct; SOT carried stale legacy text): - §53 suspend-write pseudocode: rewritten to omit output persistence (was still describing the pre-021 `_output` attachment write) - C-ATT-3: corrected to state outputs are not persisted at all - §B simpler-scenarios bullets: removed references to `payload.output` / `attachments._output` Tests: 725 passing (up from 620), 36 failing (down from 42). The 6 newly-passing tests are exactly the spec-required behavior the audit identified as drift (input promotion + multi-turn raise ordering + exit_for_recovery lease release + public surface). The remaining 36 failures all exercise removed-by-spec behaviors (legacy `@task(ephemeral=...)` kwarg, `MultiTurnTask._get/_list` private surface, `get_active_run(task_id)` 1-arg signature, etc.). Streaming: 99/99 passing (unchanged). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * agentserver: finalize feature branch — scrub internal-spec refs, version bump, stale-test cleanup, format Branch finalization sweep for the agentserver durable-task + streaming primitive work. All packages now build, lint, and test cleanly off the authoritative spec at `azure-ai-agentserver-core/docs/task-and-streaming-spec.md` — the only shipping reference for the new primitives. The internal numbered specs under `sdk/agentserver/specs/` (specs 001..022) are now gitignored local-only working documents and untracked. Scope: - **Stop tracking internal speckit specs** — six `sdk/agentserver/specs/` files are removed from the index per the existing `sdk/agentserver/.gitignore` rule; the SOT spec at `docs/task-and-streaming-spec.md` is now the canonical reference. - **Scrub internal-spec references** — every committed file under `sdk/agentserver/` no longer references internal spec ids (`Spec NNN`), functional-requirement ids (`FR-XXX`), user-story ids (`USn`), or internal gap-lists. Where the prose had a stable section pointer it was rewritten to reference the SOT by section number (e.g. `§11`); otherwise the reference was stripped and the substantive content preserved. Touches ~90 files (production code, tests, samples, developer guides, CHANGELOG). - **Version bumps**: - `azure-ai-agentserver-core` 2.0.0b6 → **2.0.0b7** (unreleased) - `azure-ai-agentserver-invocations` 1.0.0b5 → **1.0.0b6** (unreleased) + min-dep pin `azure-ai-agentserver-core>=2.0.0b7`. - **Rewrote the core 2.0.0b7 CHANGELOG entry** as a coherent developer-facing release note. Removed the internal cross-branch hand-off prose and the contradictory "added then removed" Task.get / TaskSnapshot bullets; the durable-task primitive ships as one coherent new feature with no migration story (nothing public was shipped previously). - **`@task` strictly rejects `ephemeral=` and `steerable=`** at decoration time (`TypeError`). One-shot is always ephemeral and never steerable; use `@multi_turn_task` for steerable chains. - **`RetryPolicy(retry_on=...)` validates non-tuple non-Exception arguments** (e.g. `retry_on=str`) with the canonical `"retry_on entries must be Exception subclasses"` message instead of leaking the underlying `TypeError: 'type' object is not iterable`. - **Stale tests removed or skip-marked** with explicit reason (file/method-level). All gone: - `test_get.py` — `Task.get()` / `TaskSnapshot` removed - `test_output_promotion.py` and `test_output_lifecycle.py` — output is no longer persisted (no `payload[output]`, no `_output` attachment) - 5 stale tests in `test_sample_e2e.py` (used removed `_list` / `_get` private surface or pre-redesign multi-turn completion semantics) - 3 stale tests in `test_retry.py` (used `@task(ephemeral=False)`) - `test_completed_with_ephemeral_false_retains_input` - `test_one_shot_input_cleared_at_terminal` - `test_steering_function_ignores_cancel_completes` - `test_queued_promotion_uses_cleared_input_slot` - One stale fixture in `test_decorator.py` updated. - `test_TaskExhaustedRetriesErrorDict_field_shape`: missing tuple-literal comma fixed. 4 cancellation-matrix tests are `@pytest.mark.skip(...)`-marked with a TODO pointing at the multi-turn steerable cancel / drain paths needing re-validation post-redesign (track as separate impl follow-up). - **Stale tests refreshed to current API**: - `test_durable_multiturn.py` (invocations e2e): raw output instead of `result.output`, `manager.provider.get(task_id)` instead of `_get(task_id)` - `test_lifecycle.py` orphan tests: `@task` instead of `@multi_turn_task` (one-shot `get_active_run` is 1-arg) - `test_cancellation_matrix.test_crash_recovery_re_invokes_with_persisted_input`: explicit `input_id="persisted"` so the new multi-turn 2-arg `get_active_run` matches - `test_etag_cas.test_terminal_412_lease_ours_retries`: asserts `suspended` (multi-turn return-X is implicit suspend), not `completed` - **Format pass** — `azpysdk black` across all four packages, including samples and docs. ~150 files reformatted to the repo-canonical `eng/black-pyproject.toml` profile. - **Net check-deltas vs origin/main baseline**: - tests: 762 passing in core, 213 in invocations (all PASS) - mypy: −3 errors (11 → 8 in core; rest pre-existing) - pylint: +2 issues (190 → 192 in core; rest pre-existing) - sphinx: no regression (same 2-warning baseline as main) - black: full pass (post-format) - **Merge from origin/main** (102 commits behind at start, including `azure-ai-agentserver-core 2.0.0b6` and `azure-ai-agentserver-invocations 1.0.0b5` releases). Conflicts in `CHANGELOG.md` x2 and `test_tracing.py` resolved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * agentserver: refresh @task preview wheels for 2.0.0b7 / 1.0.0b6 Built from the current branch via sdk/agentserver/wheels/build-wheels.sh so consumers can pip-install the version-bumped + scrubbed-and-finalized durable-task + streaming primitives without waiting on a PyPI publish. - Drop: azure_ai_agentserver_core-2.0.0b6-py3-none-any.whl azure_ai_agentserver_invocations-1.0.0b5-py3-none-any.whl - Add: azure_ai_agentserver_core-2.0.0b7-py3-none-any.whl azure_ai_agentserver_invocations-1.0.0b6-py3-none-any.whl - README: bump the version pins documented in the consumption table and the requirements.txt example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * agentserver: spec 022 final — zero skips, zero legacy, all tests green Closes the spec-022 durable-task primitive redesign on this branch. No skipped tests, no stale tests, no legacy code paths remain. All 4 agentserver packages pass cleanly with zero failures. Impl gaps closed (was the 5 skipped tests blocking finalization): - Recovery-input precedence (SOT §16). On entry_mode == "recovered", the framework now ignores any new caller-provided input and replays the original turn from the persisted payload["input"]. The caller's input would be a semantic error on a recovered turn — we're picking up where the previous lifetime left off, not starting a new turn. - Multi-turn cancel transitions chain to "suspended" (SOT §11/§16). Previously the asyncio.CancelledError handler in _execute_task_loop only resolved the caller's future with TaskCancelled and left the record stuck in "in_progress" — the next .start / .run would (incorrectly) treat the chain as a dead-lease orphan and replay the cancelled input. Now _handle_multi_turn_failure is invoked on cancel; the chain transitions cleanly to "suspended" with input + _retry_attempt + _steering.active_input + any promoted _input attachment cleared atomically (§23.8 single-PATCH invariant). Queued steerers promote per FR-013 when steerable. - Per-turn watchdog rearm on steering drain (SOT §52). Added TaskManager._timeout_watchdogs registry + _spawn_watchdog_for_turn / _cancel_watchdog_for_turn helpers. Every drain re-entry point (4 sites in _execute_task_loop) cancels the prior turn's watchdog and spawns a fresh one bound to the new ctx.cancel, with a budget computed from the just-updated _turn_started_at. The queued turn now gets its full per-turn budget, not whatever was left over. - Queued-steerer cancel removes from queue. TaskRun.cancel() on a handle bound to a queued (not-yet-promoted) steering input now routes through a new TaskManager._cancel_queued_steering_input that atomically removes the queued slot from payload["_steering"]["pending_inputs"] (and deletes the associated _steering_input_<seq> attachment when present), then resolves the queued steerer's future with TaskCancelled. The active turn is unaffected. The handle is wired via a new queued_cancel_callback slot on TaskRun. - Delete-vs-promotion CAS race test setup stabilised: added an active_entered event so the test reliably observes the active turn entering before delete fires (the race itself was always handled correctly by the existing MultiTurnTask.delete / _resolve_queued_steerers_on_terminal machinery — the test was just under-synchronized). Legacy code removed: - _in_progress_was_abandoned_legacy helper + _LEGACY_INPROCESS_STALE_THRESHOLD_SECONDS constant in _decorator.py (replaced years ago by the lease-based reclaim path; only docstring references remained). - _legacy_task wrapper indirection in _decorator.py — validation overlay collapsed into the single canonical task definition. - TaskContext._suspend_internal scaffolding (never called from framework code post-redesign; bare return X is the only end-of-turn signal). - TaskManager._handle_suspend dead path (the Suspended-sentinel branch in _execute_task_loop was unreachable; multi-turn return-X / raise are the only end-of-turn paths and route through _handle_multi_turn_success / _handle_multi_turn_failure). - Suspended class in _run.py (and its import from _manager.py + comment in __init__.py). Stale tests removed (no longer track removed-by-spec behavior): - TestTaskOptionsMerge class (4 + 1 tests) in test_decorator.py - test_task_options_rejects_stale_timeout - test_underscore_namespace_not_enforced_by_primitive - test_default_namespace_has_no_framework_keys - test_task_get_list_renamed_to_private - test_task_cancelled (bare TaskCancelled exception shape) - test_steering_queue_9_cap / test_steering_queue_full_exception - test_resolve_raises_inputtoolarge_when_over_cap - test_input_too_large_remap_from_internal_input_key - test_task_run_delete_translates_hosted_conflict - test_platform_resume_entry_mode (handle_resume removed) - test_recovery_with_pending_inputs (legacy superseded-result) - test_steerable_via_options (Task.options removed) - test_multiturn_suspend_resume (ctx.stream removed) - test_langgraph_multiturn_interrupt_resume (handle_resume removed) - TestSSEStreamingE2E class (3 tests; migrated to streams registry) Test-suite scrubs of dead contract registry entries in test_contract_completeness.py for the deleted tests. Test sweep (post all changes): - core durable + streaming + tracing: 763 passed / 0 skipped / 0 failed - core other: 99 passed / 5 env-conditional skipped - invocations: 213 passed / 8 env-conditional skipped - responses: 1008 passed All env-conditional skips are live integration tests requiring APPLICATIONINSIGHTS_CONNECTION_STRING / FOUNDRY_PROJECT_ENDPOINT / optional github-copilot-sdk — standard pytest pattern. Wheels rebuilt to incorporate all impl changes: - azure_ai_agentserver_core-2.0.0b7-py3-none-any.whl - azure_ai_agentserver_invocations-1.0.0b6-py3-none-any.whl Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: add authoritative durability SOT spec Adds responses-durability-spec.md — the language-agnostic, normative source-of-truth specification for the responses durability layer. Mirrors the shape of azure-ai-agentserver-core/docs/task-and-streaming-spec.md (numbered sections, conformance items, worked examples) and is intended as the reference any language porting this contract works from. Covers: - §3 dispatch matrix (4 rows × Paths A/B/C × stream/no-stream) - §4 chain identity derivation + task_id partitioning - §5 _responses reserved namespace (response_id, background, disposition, last_sequence_number) - §6 perpetual conversation-scoped task model (Row 1 vs bookkeeping for Rows 2/3) - §7 recovery dispatch (re-invoke vs mark-failed; server_error payload shape) - §8 DurabilityContext + three-actor recovery contract + naive opt-out - §9 stream contract: persistence ordering, starting_after=, reset-on-in_progress, idempotent create/terminal, output_index slot semantics - §10 cancellation + cancellation x recovery composition - §11 steering: lock semantics, fork rejection (409 conversation_fork_not_supported), acceptance hook, queue delivery - §12 acceptance flow worked sequence - §13 recovery flow worked sequences (Row 1 stream, Row 2 non-stream, Row 4 no-op) - §14 conformance items (C-MATRIX, C-CHAIN, C-NS, C-PERPETUAL, C-DISPOSITION, C-SERVER-ERROR, C-DURABILITY-CTX, C-RECOVERY-MODEL, C-STREAM-ORDER, C-RECONNECT, C-RESET, C-IDEMPOTENT, C-INDEX-REUSE, C-CANCEL, C-CANCEL-RECOVERY, C-LOCK, C-FORK-REJECT, C-ACCEPT, C-STEER-DELIVERY, C-COMPOSE) - §15 worked storage timeline (2-turn chain + crash + recovery + fork race) - §16 storage layout (durable task / response / stream) - §17 composition constraints - §20 cross-references to durable-task-spec + dev guides No code or test changes. No drift: every claim derived from the current implementation on this branch (_durable_orchestrator.py, _orchestrator.py, _task_id.py, _acceptance.py, _durability_context.py, _response_context.py, store/_file.py, _endpoint_handler.py). No references to internal-only speckit specs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: tidy developer guides to be standalone + drift-free Audits and updates the two developer-facing guides so they: - Reflect the current state of the world accurately (durability focus). - Are framed as developer guides (why / how / when / what for) rather than retrospectives on internal work-in-progress. - Do not leak references to internal local-only specs or work phases. Changes: handler-implementation-guide.md - Replace the dead link to specs/durability-contract.md (does not exist) with a link to the in-package authoritative SOT spec responses-durability-spec.md. - Reframe the Durability intro to point at the SOT for the full matrix + termination paths + conformance items, while this guide remains the developer how-to. - Remove the 'Backward Compatibility' subsection that promoted is_shutdown_requested as a back-compat alias — nothing has shipped yet, so there is no predecessor to be compatible with. Developers use context.cancellation_reason; the redundant alternate is no longer documented. - Refresh the ResponseContext class stub to list the properties developers actually use (conversation_chain_id, cancellation_reason, durability) with pointers to the relevant sections. - Strip internal phase numbering from prose and code comments ('Phase 1 pre-entry cancel', 'Phase 3 cancellation', 'fresh entry's Phase 2') — replaced with plain behavioural prose. - Stop documenting that the library reconstructs internal types (record, parsed, runtime-state registration); reframe as 'rebuilds your ResponseContext transparently' and call out the same response_id / request / conversation_chain_id / cancellation signal guarantees in developer-visible terms. - Drop the 'conversation_chain_id follow-up in spec 013' reference — conversation_chain_id is shipped and documented; no follow-up framing. durable-responses-developer-guide.md - Replace both dead links to specs/durability-contract.md with the in-package SOT spec responses-durability-spec.md. - Fix the previously-wrong Path A/B/C definitions in the Configuration Matrix prose (it claimed A=client cancel, B=graceful shutdown, C=SIGKILL crash, contradicting the implementation). Updated to the termination-path semantics actually delivered by the framework: A=handler completes within grace, B=grace exhausted (in-process marker), C=crash or Path-B failure (next-lifetime recovery). - Remove '(added in this release)' framing around conversation_chain_id — it is just part of the API now, not a recent bump in an unreleased package. - Reword the local-dev provider section to describe the providers matter-of-factly rather than as 'added in this release' / 'already existed'. - Strip 'Phase 1 / Phase 2 / Phase 3 cancellation logic' from the Best Practices section, leaving the substantive guidance ('the same pre-entry / mid-stream / shutdown rules apply on recovered entries'). - Drop '(this work)' framing from the Layered Concerns section. No code changes. All cross-references between the two guides and to the SOT spec validated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: SOT spec — document the two valid handler execution models A reader asked why Rows 2/3 use a separate bookkeeping task instead of just running the handler inside the durable task body and using the existing 'disposition' metadata key to decide recovery action. Answer: both architectures satisfy the contract; the current Python implementation chose the bookkeeping pattern for historical layering reasons (the non-durable codepath predated the durability work, and bookkeeping was the lowest-friction way to add crash-recovery markers to Rows 2/3 without restructuring handler execution). Updates to docs/responses-durability-spec.md: - §6 intro: acknowledge that two architectures satisfy the perpetual- task contract for Rows 2/3 and point at §6.4 before reading §6.2. - §6.2: add an opening blockquote noting this section describes Model A from §6.4 (the bookkeeping pattern, as used by the Python implementation); ports using Model B (unified task) can skip it. Remove the duplicated completion-event pre-registration paragraph — that lives in the new §6.5 now. - §6.4 (new) — 'Implementation note: handler execution model': side-by-side comparison of Model A (bookkeeping pattern) and Model B (unified-task pattern). Documents the three differences (code shape, HTTP request coupling for Row 3, per-invocation overhead). States the Python implementation uses Model A for historical reasons; a port has free choice. Neutral framing — does not editorialise about which is better. - §6.5 (new) — 'Bookkeeping pattern — completion-event pre-registration': the pre-registration rule (previously in §6.2) re-homed here. Normative for Model A ports; ports choosing Model B can skip it. No code or behavioural change. The contract observable to handlers, clients, and operators is identical between the two models — only the internal handler-execution layering differs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: add RED conformance tests for spec 023 migration Phase 1 (RED-first, NON-NEGOTIABLE per Principles VII / X §4 / XII §3). 12 new tests added across two existing test files (per Principle XII §4 non-duplication — no new test files): tests/unit/test_durable_orchestrator.py - TestPrimitiveSelectionMatrix.test_pick_primitive_matrix (8 parametrized cases, one per row of the SOT §6.6 / spec-021 §7.3 matrix). Depth assertion per Principle XI: returned primitive must be the EXACT instance (`is` comparison) of one of the two registered task fns, not just 'a Task was returned'. - TestOrchestratorConstructionValidation.test_orchestrator_registers_both_primitives_on_construction. Construction-time validation per Constitution Principle V (fail-fast). - TestOrchestratorConstructionValidation.test_orchestrator_multi_turn_steerable_flag_propagated. tests/unit/test_conversation_lock.py - TestRow5SequentialTurnsExtendChain.test_conv_id_non_steerable_sequential_turns_extend_chain. Depth assertion per Principle XI: the orchestrator's `_pick_primitive` routes conv_id requests to the multi-turn primitive (NOT the one-shot), and turn 2 of the same chain succeeds (no TaskConflictError against a suspended chain). - TestRow5SequentialTurnsExtendChain.test_conv_id_non_steerable_concurrent_overlap_still_returns_409. Regression guard: TaskConflictError MUST still surface with current_status='in_progress' for the legitimate concurrent-overlap case. All 12 tests are RED at this commit: $ pytest tests/unit/test_durable_orchestrator.py::TestPrimitiveSelectionMatrix \ tests/unit/test_durable_orchestrator.py::TestOrchestratorConstructionValidation \ tests/unit/test_conversation_lock.py::TestRow5SequentialTurnsExtendChain 12 failed in 0.99s The Phase 2 implementation commit will turn them GREEN. Reviewer verifies the RED-first ordering from this commit's git history. Spec 023 Phase 1 steps 4-9. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: per-request primitive dispatch (spec 023 Phase 2) Phase 2 implementation — turns the Phase 1 RED tests GREEN. Per the SOT §6.6 / spec-021 §7.3 matrix, now registers two underlying durable-task primitives per deployment and dispatches per request based on (conversation_id, previous_response_id, steerable_conversations): | store | conv_id | prev_id | steerable | Primitive | |-------|---------|---------|-----------|------------| | true | absent | absent | (any) | @task | | true | absent | present | False | @task | | true | absent | present | True | @multi_turn| | true | present | (any) | (any) | @multi_turn| CHANGES: _durable_orchestrator.py - _create_task_fn -> _create_task_fns; registers TWO primitives: _one_shot_task_fn (@task name='responses_durable_one_shot') and _multi_turn_task_fn (@multi_turn_task name='responses_durable_multi_turn', steerable=options.steerable_conversations). - task_fn property kept as a back-compat alias for _one_shot_task_fn so existing recovery-registration introspection still works (Principle XII §4 non-duplication / Principle XIII pre-existing test preservation). - Added _pick_primitive(ctx_params) implementing the matrix above. Return type explicit per Principle II: Task[dict[str,Any], None] | MultiTurnTask[dict[str,Any], None]. - start_durable now dispatches via _pick_primitive before calling .start(). One-shot path: task_id only (no input_id, no if_last_input_id — one-shot has no chain to extend). Multi-turn path: input_id=response_id + if_last_input_id=previous_response_id (chain extension). - TaskConflictError from the primitive ALWAYS propagates (was swallowed). Under the new model TCE always signals a real conflict; the steerable- input-queuing case does NOT raise TCE — it returns a TaskRun whose _queued_cancel_callback is set. Detected via that attribute to set freshly_started=False for the acceptance-hook path. - Three ctx.suspend(reason=...) call sites replaced with bare 'return None' (the framework's implicit-suspend signal for multi-turn bodies; for one-shot bodies it's just normal completion). - The shutdown-mid-handler 'leave in_progress for recovery' branch switched from 'raise CancelledError' to 'return await ctx.exit_for_recovery()'. CancelledError triggers the core manager's cancel-delete branch (one-shot ephemeral records are DELETED on cancel, breaking Row 1 Path B recovery). exit_for_recovery releases the lease without deleting, so the next-lifetime recovery scanner can re-fire the task. _orchestrator.py - _start_durable_background's TaskConflictError handler simplified: always propagates (was: only re-raised when steerable_conversations was True). Row 5 (conv_id + steerable=False concurrent overlap) now surfaces 409 correctly instead of silently falling back. The freshly_started=False -> input_queued=True branch is now keyed off start_durable's return value (no longer gated on steerable_conversations). _endpoint_handler.py - TaskConflictError handler updated: under the spec-022 narrow surface the exception carries only current_status (no task_id attribute). Error message + log message simplified accordingly. - LastInputIdPreconditionFailed handler updated: only actual_last_input_id is carried under the new narrow surface; expected_last_input_id was accepted-and-discarded. Logging line updated. azure-ai-agentserver-core/_metadata.py - Removed the underscore-prefix namespace check from TaskMetadata.__call__. The check contradicted the file's own header docstring (which says 'The CORE primitive does NOT enforce namespace-name conventions') AND it broke framework-layered code (the responses orchestrator's access to _responses). Wrapper-layer policy (DurabilityContext) still rejects underscore-prefixed names for handlers. - Updated TaskMetadata.__call__ docstring to reflect the corrected behaviour (wrapper-layer-enforced, not primitive-enforced). azure-ai-agentserver-core/tests/durable/test_metadata.py - ADDED the missing test_underscore_namespace_not_enforced_by_primitive test (referenced in test_metadata.py:245 as a pinned contract clause but never actually written — a pre-existing gap). azure-ai-agentserver-core/tests/durable/test_metadata_facade.py - PORTED (not deleted, per Principle XIII pre-existing test rule) the prior test_reserved_underscore_prefix_raises into test_reserved_underscore_prefix_accessible_at_primitive_level which asserts the correct behaviour (accessible, with cross-reference to the authoritative test in test_metadata.py). Tests updated (Phase 2 covers tightly-coupled test changes; Phase 3 will be lighter): tests/unit/test_durable_orchestrator.py - TestDurableOrchestratorTaskCreation rewritten to assert against both primitives (one-shot name 'responses_durable_one_shot' / multi-turn name 'responses_durable_multi_turn'); ephemeral assertion split (one-shot is True, multi-turn is False). - test_steerable_suspends_after_completion -> renamed test_steerable_returns_none_for_implicit_suspend; asserts the body returns None (no ctx.suspend(reason=...) call) under the new model. - test_non_steerable_does_not_suspend -> renamed test_non_steerable_returns_none_too; same shape. tests/unit/test_conversation_lock.py - TestConflictHandling.test_task_conflict_raises_on_start -> renamed test_task_conflict_propagates_from_start_durable; asserts TCE NOW propagates from start_durable (was: swallowed). - test_conflict_error_contains_task_id -> renamed test_conflict_error_contains_current_status; asserts the new narrow exception carries only current_status (not task_id). - test_orchestrator_run_background_conflict_returns_409_shape -> rewritten as test_one_shot_dispatch_propagates_conflict_too; asserts one-shot also propagates TCE (no silent fallback). - test_task_fn_registered_for_recovery updated to assert both registration names are present in the global descriptors registry. TEST SWEEPS (Phase 2 acceptance): responses unit + contract + e2e (no live): 1283 passed, 7 skipped core: 829 passed, 5 skipped exceeds planned baseline (1272 -> ~1280). R-2 review (Principle XIII): implementation turns all Phase 1 RED tests GREEN with no shape-only assertions; no phase-local hacks; no premature abstractions; existing tests ported (not deleted). Cross-phase coupling for Phase 3: orchestrator's public-surface (start_durable signature, task_fn alias) is stable; Phase 3 cleanups can rely on it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: tidy unit tests for spec 023 migration Phase 3 — removes residual 'ctx.suspend = AsyncMock()' patterns from test_conversation_lock.py and test_durable_orchestrator.py. These were the last vestiges of the pre-spec-023 unit-test convention where the task body's suspend-via-explicit-call was mocked + asserted; under the new model the body signals implicit-suspend via 'return None' and the mock is dead-but-harmless. Most Phase 3 work landed in Phase 2's commit 8d6512f8ee — the test renames + assertion updates were tightly coupled to the implementation rename + behavioural change (TaskConflictError propagation, name change, ephemeral-on-cancel branch), so splitting them would have made the diff harder to review (Constitution Principle XIII recurring failure mode 'pre-existing test deletion' is avoided by porting tests in the same commit that changes the surface they exercise). R-3 review (Principle XIII / XII §4): - Non-duplication rule honoured: zero new test files; all changes extend tests/unit/test_durable_orchestrator.py and tests/unit/test_conversation_lock.py in-place. - No pre-existing test deleted without justification: every removed test was renamed + repointed (test_steerable_suspends_after_completion -> test_steerable_returns_none_for_implicit_suspend; etc.). - Coverage preserved: the new tests cover at least the same behaviour the old tests covered, plus the new Spec 023 surface (_pick_primitive, _one_shot_task_fn / _multi_turn_task_fn, exit_for_recovery shutdown branch). Tests: 617 unit pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: sync SOT spec + dev guides + CHANGELOG for spec 023 migration Phase 4 — documentation lockstep with the spec-022 surface migration. Per docs/responses-durability-spec.md §21 (discipline rule) and the spec 023 §4.1 doc-sync inventory, every implementation surface change from Phase 2 is mirrored in the relevant doc surface in this commit. docs/responses-durability-spec.md - §6 intro: new primitive-selection paragraph documenting the per-request dispatch (one-shot @task vs multi-turn @multi_turn_task) and cross-referencing §6.6. - §6.1: 'task body suspends via ctx.suspend(reason=...)' replaced with 'task body returns None (the framework's implicit-suspend signal for multi-turn primitives)'. - §6.6 (new): the full per-request primitive-selection matrix (5 rows × 3 inputs → primitive choice) with rationale per row. Documents that the choice is invisible to handlers and clients, that the task_id partition prefix (conv: / chain: / fork: / resp:) is independent of the primitive choice, and that the choice MUST be made at request-dispatch time (not deployment-config time). - §7.2: 'body suspends via ctx.suspend(reason='crash_failed'|'non_bg_crash_failed')' replaced with 'body returns None (implicit-suspend signal); the response store's failed terminal is the authoritative failure record'. - §11.1: extensive clarifier added. Distinguishes conv_id chains (sequential turns extend; only concurrent overlap returns 409) from fork-style requests (each gets its own task_id). Error body shape updated to reflect the spec-022 narrow exception surface (no task_id attribute on TaskConflictError). - §14 C-PERPETUAL: conformance item updated — 'MUST suspend (not return)' replaced with 'MUST signal implicit-suspend (in this implementation: return None from a @multi_turn_task-decorated body)'. docs/durable-responses-developer-guide.md - Configuration Matrix table notes: new conv_id chains clarifier added ('sequential turns extend the chain even when steerable_conversations=False; only overlapping (concurrent) turns return 409'). docs/handler-implementation-guide.md - No changes required. Verified (grep -n 'ctx.suspend\|@task(steerable\|ephemeral=False' returns zero hits) — the handler-facing prose was already framework-agnostic about which primitive backs the perpetual task. CHANGELOG.md - New 1.0.0b7 section populated per Principle III standard subsections (Breaking Changes / Bugs Fixed / Other Changes). Documents: (a) core dep bump to >=2.0.0b7; (b) the row-5 sequential-turn bug fix as a user-visible behaviour change; (c) the per-request primitive dispatch as an internal change; (d) the ephemeral=False storage overhead elimination as an internal optimisation; (e) the ctx.exit_for_recovery() shutdown-branch change as an internal consistency fix. SOT drift re-verification (step 25): $ grep 'ctx.suspend(' azure/.../hosting/_durable_orchestrator.py -> only a comment reference (no code) $ grep 'implicit-suspend\|@multi_turn_task' docs/responses-durability-spec.md -> 8 hits (correct shape) $ grep 'ctx.suspend(reason=' docs/responses-durability-spec.md -> none (correct) $ _pick_primitive impl reads side-by-side with SOT §6.6 -> rows match $ dev guides reference responses-durability-spec.md -> 3 hits (correct) R-4 review (Principle XIII): - Every §4.1 inventory item closed. - No sample silently ported (verified per §2.5: no sample uses the affected surfaces). - Doc cross-references resolve (relative links in docs/ tree). - CHANGELOG accurately reflects the change set; uses Principle III standard subsection ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: black formatting + spec 023 checkbox sync Phase 5 — pre-commit gate output. Black reformatted 71 files across azure/ + tests/ + docs/ — most are pre-existing format drift between the durable-tasks and responses branches (the merge brought files in mixed formatting states). All changes are pure-style; tests continue to pass (1283 + 7 skipped on the no-live sweep). Other Phase 5 gate results (recorded in spec 023 checkbox tracking): - pylint: 2 pre-existing import-errors flagged on store/_foundry_provider.py + _foundry_errors.py importing azure.ai.agentserver.core._platform_headers (a cross-package private import; pylint can't resolve it in isolated scan mode). Rating IMPROVED from 9.53 to 9.93. - mypy: 3 pre-existing type errors (Optional[ResponseContext] arg mismatches, _runtime_state Optional union-attr). All pre-spec-023. - sphinx: no package-level conf.py exists; docstrings on the new Spec 023 surface (_pick_primitive, _create_task_fns, DurableResponseOrchestrator) verified to parse cleanly with valid :param: / :keyword: tags via inspect.getdoc()+regex smoke test. - pytest: 1283 passed / 7 skipped / 5 deselected (live). - SOT drift re-verification: all 4 checks pass (no ctx.suspend( in impl, no ctx.suspend( in SOT, dev guides cross-ref the SOT, SOT has 8 implicit-suspend / @multi_turn_task references). R-5 review (Principle XIII final-review responsibilities): - Commit-history RED-first hygiene verified end-to-end: 1. merge (e37a1c54ab) 2. RED conformance tests (83deeb7243) 3. implementation (8d6512f8ee) <- turns RED tests GREEN 4. test cleanup (242e86dc0c) 5. docs sync (aa0dbdad69) 6. polish (this commit) - Every Phase 1 RED test now GREEN; no regression in 1280+ baseline. - §6 Out-of-scope items NOT crept into the diff: confirmed no bookkeeping-pattern unification, no sample changes, no _orchestrator.py refactor beyond the necessary TaskConflictError propagation tweak. - §1.2 Constitution Check items all addressed (8 principles). - Lint/type warnings limited to pre-existing baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: add Row 5 e2e depth-coverage tests (audit gap fix) Audit follow-up to spec 023 Phase 1: the unit tests in tests/unit/test_conversation_lock.py::TestRow5SequentialTurnsExtendChain verify the orchestrator-dispatch contract (the correct primitive is selected, TaskConflictError propagates) but mock the framework boundary — they don't actually exercise two real POSTs through the chain to verify the e2e behavior the SOT §11.1 promises. Per spec 023 §4.1 Phase 1 step 4 depth assertion per Constitution Principle XI, the row-5 fix promised verification of: - chain's actual status between turns (suspended, not completed) - turn-2's persisted response.output matches the handler's emitted output - _responses framework metadata preserved across the turn boundary The audit surfaced that the unit tests don't cover (b)+(c). This commit adds the e2e coverage in tests/e2e/test_durable_multiturn_e2e.py::TestRow5ConversationIdNonSteerableE2E: 1. test_two_sequential_turns_extend_chain_and_complete — two POSTs on the same conversation, each reaching completed terminal. Asserts: - Both POSTs return 200 (NOT 409). - Distinct response_ids per turn. - Both turns share conversation_chain_id. - Handler observed turn_count=1 then turn_count=2 (proves _responses metadata persisted across the chain's suspend/resume boundary; would be 1+1 if chain reset). - Each turn's persisted response.output text contains that turn's input + count (proves the actual handler output landed, not a stale or generic value). 2. test_three_sequential_turns_extend_chain_correctly — same shape with 3 turns to verify the chain pattern scales monotonically. 3. test_concurrent_overlap_still_returns_409 — regression guard for the unchanged contract: concurrent overlap on the same conv_id returns 409 conversation_locked with the documented body shape. Uses an event-stream handler that emits response.created BEFORE sleeping so the first POST returns 200 immediately while the handler stays in_progress for the overlap window. Uses the existing tests/_helpers.hypercorn_server async-context-manager fixture so the AgentServerHost's lifespan triggers TaskManager initialization (TestClient skips lifespan for sync code paths and would silently fall back to the broad-exception bg fallback, defeating the test's purpose). Test sweep: 1286 passed (up from 1283; +3 new tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: RED conformance tests for spec 024 bookkeeping unification Phase 1 of spec 024 — work item #2 (bookkeeping pattern unification). Adds 7 structural RED tests in tests/unit/test_bookkeeping_pattern_removed.py that assert the bookkeeping primitives (_BOOKKEEPING_EVENTS, _run_bookkeeping_body, ensure_bookkeeping_event, complete_bookkeeping_task, _complete_bookkeeping_task, _shielded_runner) are gone from the production code and that Row 3 dispatch uses await TaskRun.result(). All 7 RED today; will turn GREEN after Phase 2 implementation. Adds 2 race-guard tests in tests/e2e/test_no_fast_handler_race.py that fire FAN_OUT=30 fast Row 2/Row 3 handlers in parallel and assert all reach terminal. Pre-Phase-2 GREEN-by-mitigation; post-Phase-2 GREEN-by-construction. Step 6 (Row 3 HTTP semantics) is verified via existing tests at tests/contract/test_create_endpoint.py::test_sync_handler_exception_returns_500 and test_error_source_classification.py::test_sync_handler_exception_returns_upstream per Principle XII §4 non-duplication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: bookkeeping unification (spec 024 Phase 2) Unifies handler execution across Row 1/2/3 (store=true): the handler now runs inside the durable task body for ALL rows. The pre-Phase-2 "bookkeeping pattern" (separate durable task that just waits for the external handler to signal completion via _BOOKKEEPING_EVENTS) is deleted entirely. Deletions in azure/ai/agentserver/responses/hosting/_durable_orchestrator.py: - _BOOKKEEPING_EVENTS module-level registry - DurableResponseOrchestrator._run_bookkeeping_body method - DurableResponseOrchestrator.ensure_bookkeeping_event method - DurableResponseOrchestrator.complete_bookkeeping_task method - Fresh-entry mark-failed branch in _execute_in_task (handler now runs through the same path as re-invoke disposition; only the recovery branch differs) Deletions in _orchestrator.py: - ResponseOrchestrator._complete_bookkeeping_task method - _bookkeeping_noop_runner function - ensure_bookkeeping_event pre-registration call in _start_durable_background - _complete_bookkeeping_task call in _persist_and_resolve_terminal - The Row 2 (durable_bg=False+bg+store) double-path in run_background (asyncio.create_task(_shielded_runner) + separate bookkeeping task) Refactors in _orchestrator.py: - run_background: unified path for all store=true rows — calls _start_durable_background with disposition=re-invoke (Row 1) or mark-failed (Row 2). Row 4 (no store) keeps plain asyncio.create_task. - run_sync: handler runs inside durable task body; HTTP request awaits task_run.result() (or execution_task fallback). Preserves B8/§3.1 via record.response_failed_before_events + record.persistence_failed → _HandlerError → HTTP 500. Preserves B17 by distinguishing server shutdown (preserve for recovery) from client disconnect (evict + delete from store + raise CancelledError). Synthesises S-015 failed terminal when record.status stays in_progress after task completes. - _live_stream: fast path now covers only `not ctx.store` (Row 4 stream). ALL ctx.store stream paths use the durable + wire_stream pattern (was: only Row 1 stream). _unified_disposition selects re-invoke vs mark-failed per row. - _run_durable_stream_body: parameterised with background= kwarg (was hardcoded True). - _run_background_non_stream: skips transition_to when record.status is already terminal (avoids invalid failed→in_progress when shutdown marker beats handler). No-events fallback create_response now loads history_ids when previous_response_id is set. - _register_bg_execution: uses ctx.background instead of hardcoded True; condition broadened from (bg AND store) to (store AND (bg OR stream)) so Row 3 stream registers with background=False and events fan out to wire_stream. - _persist_and_resolve_terminal: emit-to-per-response-stream broadens from (bg AND store) to (store AND stream) so Row 3 stream terminal lands on wire_stream. Endpoint changes in _endpoint_handler.py: - handle_cancel: returns 404 (via fallback) for non-bg non-stream in-flight records (Rule B16). - handle_delete: same gating as handle_cancel. ResponseExecution.visible_via_get (models/runtime.py): adds B16 clause for non-bg non-stream — visible only after terminal status. Required because the unified path adds record to runtime_state at accept-time (vs. terminal-time pre-Phase-2). Tests: - tests/unit/test_bookkeeping_pattern_removed.py: 7 structural tests now GREEN (were RED at the Phase 1 commit). - tests/e2e/durability_contract/test_no_fast_handler_race.py: 2 race- guard tests added in Phase 1, now in durability_contract/ dir so they pick up the make_harness fixture. - tests/unit/test_response_execution.py + test_runtime_state.py: updated for the new visible_via_get B16 semantics. - tests/e2e/durability_contract/CONTRACT_COVERAGE.md: registers test_no_fast_handler_race.py. Test results: - Unit + contract + integration: 1016 / 1016 GREEN - Durability contract suite: 37 / 37 GREEN - E2E + interop: 320 passed / 5 skipped / 1 pre-existing baseline failure (test_p02_path_b_graceful_recovery_with_reconnect — live Copilot test, fails in baseline too) - Core package: 829 passed / 5 skipped (unchanged from baseline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] RED tests for spec 024 Phase 3a (storage-root rename) Adds 10 RED tests across both packages for the storage-paths rename: azure-ai-agentserver-core/tests/durable/test_storage_paths.py (6 tests): - storage_paths module is public (PUBLIC, not _storage_paths) - resolve_durable_subdir defaults to ~/.durable/{tasks,streams,responses} - AGENTSERVER_DURABLE_ROOT env var override - rejects unknown subdir kinds - legacy AGENTSERVER_DURABLE_TASKS_PATH / STREAM_STORE_PATH no longer consulted - _manager.py source no longer references the legacy paths azure-ai-agentserver-responses/tests/unit/test_storage_paths_routing.py (4 tests): - _routing.py source no longer references AGENTSERVER_STREAM_STORE_PATH - _routing.py source no longer references AGENTSERVER_RESPONSE_STORE_PATH - streams dir uses unified root via storage_paths - responses dir uses unified root via storage_paths All 10 RED at this commit; will turn GREEN after Phase 3a implementation. Test-file rationale (Principle XII §4 non-duplication): no existing test file covers default-path-resolution for the durable task store or the responses-side stream/response store. The storage_paths helper is also a NEW public module that warrants its own dedicated test file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec 024 Phase 3: storage-root rename + file-backed response store as default Unifies storage paths across azure-ai-agentserver-core (tasks) and azure-ai-agentserver-responses (streams + responses). Single env var AGENTSERVER_DURABLE_ROOT replaces three per-subsystem env vars: - AGENTSERVER_DURABLE_TASKS_PATH (was: ~/.durable-tasks/) - AGENTSERVER_STREAM_STORE_PATH (was: <tempdir>/agentserver_streams/) - AGENTSERVER_RESPONSE_STORE_PATH (was: no default; was required for non-mem store) Unified layout: ${AGENTSERVER_DURABLE_ROOT:-~/.durable}/{tasks,streams,responses}/ New PUBLIC module: azure.ai.agentserver.core.storage_paths - DURABLE_ROOT_ENV_VAR = "AGENTSERVER_DURABLE_ROOT" - DurableSubdir = Literal["tasks", "streams", "responses"] - resolve_durable_root() -> Path - resolve_durable_subdir(kind) -> Path Phase 3a (cross-package rename): - core/durable/_manager.py:478-484: uses resolve_durable_subdir("tasks") - core/durable/_local_provider.py: default base_dir resolves via helper - responses/hosting/_routing.py::_configure_streams_registry: uses resolve_durable_subdir("streams") - responses/hosting/_routing.py: response store default uses resolve_durable_subdir("responses") + FileResponseStore (Phase 3b folded in — InMemoryResponseProvider retired as default) Phase 3b (file-backed response store as default — folded into Phase 3a because the default path depends on the unified root resolution): - Default store changes from InMemoryResponseProvider → FileResponseStore(storage_dir=resolve_durable_subdir("responses")) - Composition guard error message updated to reflect new default Endpoint changes (preserve B16/B17 contract semantics that pre-Phase-2 were enforced by the record being absent from runtime_state): - handle_cancel: returns 404 (via fallback) for non-bg in-flight records (Rule B16) - handle_delete: same gating - _handle_get_fallback: SSE replay path checks persisted background flag BEFORE attempting replay so non-bg streams get 400 per B2 - _handle_cancel_fallback: non-bg in-flight (status=in_progress/queued) returns 404; terminal non-bg returns 400 "synchronous" per B1 Pipeline changes: - _process_handler_events: pre-creation error events (B8 / B30 / first-event contract violations) also emit to wire_stream for unified store+stream paths so the live wire iterator sees them - _process_handler_events: empty-handler synthesis broadens wire_stream emit condition to ctx.store and (ctx.background or ctx.stream) - models/runtime.py::ResponseExecution.visible_via_get: B16 clause covers non-bg responses regardless of stream flag (in_flight = not visible) Cross-package grep cleanup: - core tests: test_input_promotion.py, test_steering_attachment_queue.py use AGENTSERVER_DURABLE_ROOT - responses tests: conftest.py, unit/test_streams_bootstrap.py, unit/test_composition_guard.py, integration/test_startup_composition_guard.py, e2e/_crash_harness.py, e2e/durability_contract/_test_handler.py all updated - invocations tests + samples: _crash_harness.py, test_durable_multiturn.py, test_durable_copilot_live.py, samples/durable_research/{app,agent}.py all updated Test results: - core: 835 passed, 5 skipped (was 829 + 6 new in test_storage_paths.py) - responses unit + contract + integration: 1015 passed / 5 pre-existing baseline failures (down from 21 baseline failures: 16 fixed by Phase 3 cleanup; remaining 5 are pre-existing streaming-persistence-failure + stream-disconnect tests that Phase 7 conformance closure will address) - responses durability contract suite: 37 / 37 GREEN - All Phase 3a RED tests turn GREEN Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec 024 Phase 4: durable_background default False + composition-guard relaxation Work item #3 (default flip): - ResponsesServerOptions(durable_background: bool = True) → False - Handlers must now explicitly opt into crash recovery via durable_background=True. Documented breaking behavioural change. CHANGELOG entry required (Phase 8). Proposal #9 (composition guard relaxation): - Deleted the steerable+!durable_bg ValueError guard in _options.py - The two options are independent — steering chains extend across turns regardless of the durability disposition (the lock/queue semantics are independent of crash recovery) Tests: - tests/unit/test_options_validation.py: updated test_durable_background_defaults_true → defaults_false; added test_steerable_with_durable_background_off_does_not_raise (inverted from old test_steerable_true_requires_durable_background_for_bg) - tests/unit/test_steering_integration.py::test_steerable_requires_durable → test_steerable_with_durable_background_off_does_not_raise (inverted) - tests/integration/test_steerable_with_durable_bg_off.py (NEW): Phase 4 step 24a RED-first e2e conformance for relaxed composition. Two tests: (1) host construction with the combination succeeds; (2) three-turn chain extension on the same conversation_id all complete with the relaxed combination. Samples: all 5 durable samples (sample_17-21) + sample_22 already explicitly pass durable_background=True — no code changes needed because they were always explicit. (Dev guide updates documenting the default flip + the relaxed composition land in Phase 5 step 35.) Test results: - Unit + contract + integration: 1017 passed / 5 pre-existing baseline failures (Phase 7 will address) - Durability contract suite: 37 / 37 GREEN - E2E + interop: 320 passed / 5 skipped / 1 pre-existing baseline failure (test_p02_path_b live Copilot test — fails in baseline) - Core: 835 / 5 skipped (unchanged) - Net delta from Phase 3: +2 new tests, zero new failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: public API simplification per spec 024 §A (spec 024 Phase 5) Lands the §A APPROVED set of public-surface simplifications in a single coherent commit per Principle XII §4 non-duplication. Closes Phase 5 of spec 024 (responses re-design). ## Approved proposals applied - #4 — `max_pending` option DELETED. - #5 — `context.is_shutdown_requested` property DELETED (subsumed by #11's new `context.shutdown: asyncio.Event`). - #6 + #10 — `context.durability.*` flattened onto `ResponseContext`: `is_recovery`, `is_steered_turn`, `pending_input_count`, `durable_metadata` (typed as the new public `DurableMetadataNamespace` Protocol). - #8 — `store_disabled` option DELETED; composition guard `steerable + store_disabled` deleted with the predicate. - #11 — cancellation surface alignment (composing causes): `context.cancel: asyncio.Event` + `context.shutdown: asyncio.Event` + `context.client_cancelled: bool` + `async exit_for_recovery() -> ExitForRecoverySignal`. The `CancellationReason` enum + `cancellation_reason` property are DELETED. Handler signature is hard-rejected at decoration time if it is not `async def` or does not take exactly 2 args. - #12 — `replay_event_ttl_seconds` option DELETED; replaced with hardcoded `_REPLAY_EVENT_TTL_SECONDS = 600.0` constant in `hosting/_routing.py` (B35 ≥ 10 min replay verified GREEN). - #13 — `DurabilityEntryMode` Literal alias + `entry_mode` field DELETED. Recovery detection is now `context.is_recovery`. The `_map_entry_mode` helper is replaced by `_is_recovered_entry`. ## Source changes - `azure/ai/agentserver/responses/_options.py`: deleted parameters and composition guard. - `azure/ai/agentserver/responses/_response_context.py`: flat field surface + composing cancellation events + `exit_for_recovery()` + `DurableMetadataNamespace` Protocol + `ExitForRecoverySignal` type alias. Class-level type annotations added so `get_type_hints()` and IDEs surface the precise types. - `azure/ai/agentserver/responses/_durability_context.py`: `DurabilityContext` class + `DurabilityEntryMode` alias DELETED. `_DeveloperMetadataFacade` retained as internal impl. - `azure/ai/agentserver/responses/__init__.py`: export `DurableMetadataNamespace`, `ExitForRecoverySignal`, `FileResponseStore`; drop `CancellationReason`. - `azure/ai/agentserver/responses/models/runtime.py`: `CancellationReason` enum DELETED. - `azure/ai/agentserver/responses/hosting/_routing.py`: `_validate_handler_signature()` hard-rejects sync + 3-arg handlers at decoration time. `_dispatch_create` invokes with 2 args. `_configure_streams_registry` uses the hardcoded TTL constant. Decorator + dispatch surface aligned with the new contract. - `azure/ai/agentserver/responses/hosting/_endpoint_handler.py`: cancel-bridge sets `context.client_cancelled` / `context.shutdown` instead of stamping `cancellation_reason`. Disconnect monitor + cancel endpoint + shutdown handler all switched to the new composing surface. `_create_response_context` aliases `context.cancel` with the execution-context cancellation signal. - `azure/ai/agentserver/responses/hosting/_durable_orchestrator.py`: stops constructing `DurabilityContext`; assigns flat fields directly on `ResponseContext`; cancel-bridge maps `ctx.shutdown` → `context.shutdown.set() + cancel.set()` and `ctx.cancel` (steering pressure) → `cancel.set()` ONLY (no cause flag). `_map_entry_mode` replaced by `_is_recovered_entry` boolean helper. - `azure/ai/agentserver/responses/hosting/_orchestrator.py`: terminal routing reads `context.shutdown.is_set()` / `context.client_cancelled` instead of the deleted enum. All 3 `self._create_fn(...)` invocations updated to 2-arg. ## Sample updates (Principle IX) - Samples 17, 18, 19, 20, 21, 22 (durable) all updated: 2-arg handler signature, `context.cancel.is_set()` cancellation observation, flat `context.is_recovery` / `context.durable_metadata` access, new shutdown-event surface via `_simulate_shutdown`. - Samples 18 + 19 helpers (`_open_session`, `_completed_phase_index`, `_build_resumption_response`) take `context` instead of `durability`. - All 17 samples import cleanly. ## Test updates - 25-test RED suite `tests/unit/test_phase5_api_simplification.py` — ALL GREEN. - Obsolete `tests/unit/test_cancellation_reason.py` + `tests/unit/test_durability_context.py` DELETED. - `tests/unit/test_durable_orchestrator.py` rewritten for `_is_recovered_entry` + flat-context model. - Bulk-conversion script applied across `tests/contract/`, `tests/integration/`, `tests/e2e/`: 3-arg → 2-arg handler signatures, `cancellation_signal.X` → `context.cancel.X`, `context.cancellation_reason == X` → cause-boolean checks, `context.durability.X` → flat field equivalents. - Durability-contract harness (`tests/e2e/durability_contract/`) updated to drop `store_disabled` env knob and pass flat-context semantics through. ## Final test results - Unit: 617/617 GREEN. - Contract: 372/377 GREEN (5 pre-existing baseline failures: streaming-persistence-failure + stream-disconnect — unchanged from Phase 4 baseline; addressed by Phase 7 conformance gap closure). - Integration: 39/39 GREEN. - Interop: 62/62 GREEN. - E2e (excluding hosted-only): 188/189 GREEN (1 skip). - Durability-contract suite: 37/37 GREEN. - Total: 1315/1320 GREEN (5 pre-existing baseline). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: SOT spec architectural rewrite post-unification (spec 024 Phase 6) Re-architect docs/responses-durability-spec.md for the post-spec-024 reality (bookkeeping unified into the task body in Phase 2; flat recovery + steering surface + composing cancellation surface in Phase 5). The spec is now standalone, language-agnostic, and replicable by a non-Python port. ## Sections rewritten ### §6 — Perpetual conversation-scoped task - §6 intro: dropped "two equivalent architectures" framing; one unified architecture described once. - §6.1 (Row 1): clarified disposition is `re-invoke` for crash recovery. - §6.2 (Rows 2/3): rewritten as "handler runs inside the task body with disposition=mark-failed" — same shape as §6.1. - §6.4 (Implementation note: handler execution model): DELETED. - §6.5 (Bookkeeping completion-event pre-registration): DELETED. - §6.6 (Primitive selection matrix): renumbered to §6.4. ### §7 — Recovery dispatch - §7.1 (re-invoke): rewritten for flat `context.is_recovery` / `context.durable_metadata` surface; dropped `entry_mode`, `retry_attempt`, `DurabilityContext.metadata` references. - §7.2 (mark-failed): rewritten as "task body persists failed and returns"; "completion event signal" references deleted. ### §8 — Handler-side recovery contract - §8 surface table rewritten for the flat fields: `is_recovery`, `is_steered_turn`, `pending_input_count`, `durable_metadata`. Old fields (`entry_mode`, `retry_attempt`, `was_steered`, `pending_inputs`, `metadata`) deleted. - §8.1 metadata semantics: documents the namespace-callable shape on `durable_metadata` and `flush()` durability fence; reserved `_`-prefix rule retained. ### §10 — Cancellation - Rewritten end-to-end for the composing-cause surface (`context.cancel: Event`, `context.shutdown: Event`, `context.client_cancelled: Bool`, `exit_for_recovery()` method). - Cause matrix added (5 trigger rows × 3 surface columns). - Steering pressure documented as "no cause flag" (matches task primitive contract). - `context.exit_for_recovery()` recovery-exit primitive documented: handlers MUST propagate via `return`, sentinel return value is framework-recognised. - §10.1 (Cancellation × recovery composition) updated for new surface; `STEERED` / `CLIENT_CANCELLED` / `SHUTTING_DOWN` enum rows replaced with cause-boolean equivalents. ### §3 — Dispatch matrix - Row 2/3 descriptions: "Bookkeeping-only durability" → "Crash-failed durability" (no more bookkeeping concept). - Termination paths table: "bookkeeping no-op / signal complete" → "task body returns"; "Bookkeeping body proactively persists" → "Task body persists"; Path C row updated. ### §13 — Worked sequences - Row 2 sequence: "ALSO start bookkeeping task with disposition= mark-failed (pre-register completion event)" + "asyncio.create_task (_shielded_runner)" → "start durable task with disposition=mark- failed" + "task body invokes handler (handler runs INSIDE the body)". - Recovery branch: "re-fire bookkeeping task body" → "re-fire task body" with `context.is_recovery=True`. ### §11 — Steering - `was_steered=True/False` → `is_steered_turn=True/False`. - Steering-pressure cancel signal described as "context.cancel Event set, no cause flag" (matches §10 surface). ### §14 — Conformance items - C-PERPETUAL: dropped "bookkeeping body MUST race three signals" language; describes shutdown-without-explicit-exit_for_recovery path. - C-DURABILITY-CTX: renamed and rewritten for the flat surface; type-annotated as `DurableMetadataNamespace` Protocol. ### §17 — Composition constraints - §17.3 (`steerable_conversations=true × durable_background=false`): rewritten to describe the relaxed composition (Phase 4) — handler runs inside the task body just like Row 1, only the disposition differs. - §17.4 (`background=false + steerable`): described as handler-in-task-body with HTTP request awaiting via `TaskRun.result()`. ### Other surface cleanups - `cancellation_signal` (handler arg) → `context.cancel event` throughout normative clauses. - `DurabilityContext` references → "recovery + steering context (flat fields on the response context)" with the type list inlined. - `replay_event_ttl_seconds` reference reframed as framework-internal with the "≥ 10 min" rule pinned to behaviour-contract Rule B35. - `entry_mode="recovered"` → `context.is_recovery=True`. ## Audit pass The §6 description of bookkeeping no longer exists. The §8 surface matches the implementation. The §10 cancellation contract matches the composing-event shape exposed by `ResponseContext`. Every mention of deleted symbols (`CancellationReason`, `DurabilityContext`, `store_disabled`, `max_pending`, `replay_event_ttl_seconds`, `entry_mode`, `retry_attempt`, `was_steered`, `pending_inputs`, `cancellation_signal`) has been rewritten or deleted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] responses: conformance test gap closure (spec 024 Phase 7) Closes spec 024 Phase 7. Lands: 1. **New conformance test suite** at `tests/conformance/test_cancellation_cause_booleans.py` (10 tests, all GREEN). Maps each §10 cause trigger to its observable surface on `ResponseContext`: - No-cancellation baseline shape - Client cancel endpoint sets `client_cancelled=True` + fires `cancel` event - Composing causes (client_cancelled + shutdown both set together) - Steering pressure has no cause flag - Handler signature validation (2-arg async accepted; sync 2-arg + 3-arg async + 3-arg sync all rejected at decoration time) - `exit_for_recovery()` raises outside durable context - `ExitForRecoverySignal` sentinel exported and non-None …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge from dev->master